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dass. Event or action descriptions are transported into the respective cells of the State TaWe in the form 
of descriptive strings of text This text describes the event or action, and the event source, or action 
destination. Entries in the Slate Table define the operation of the simulation and are executed directly 
by an interpreter or are compiled to generate a program of instructions for performing the simulation. 
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BACKGROUND OF THE INVENTION 

1. Field of the Invention 

5 The Invention relates to a system for and method of interactive, graphical asmputersirnulation and, in par- 

ticular, to a system and method for providing user programming of interactive, graphic and dynamic applica- 
tions through pictorial means. 

2. Description of the Related Art 

Digftal computers have long been used to perform complicated and tedious numeric calculations and data 
processing. With increased processing speed and memory capability, the roles of digital computers have ex- 
panded to include real-time simulations of mechanical, electrical, and ether complex systems. 

At first, software for performing digital computer simulation required advanced programming skills, the 

15 programmer coding the instructions in a low level laig uage such as assembly language or machine code. This 
was done to provide the necessary interface between operator inputs and graphic objects displayed on an 
output device such as a graphic display or video monitor, With the development of high level languages, it be- 
came feasible to construct program generator systems to assist In the development of programs for performing 
the simulation. An example of suchasimulat on system is t he Virtual Applications Prototyping System (VAPS) 

20 commercially available from the assignee of the subject application. The system Is disclosed in system doc- 
umentation publications including "VAPS: Conceptions! Overview"; "VAPS: Reference Guide*; "VAPS: Pro- 
grammers^ Guide"; and "VAPS: installation Guide and Options" al by Virtual Prototypes (1991) and incorpo- 
rated herein by reference. 

The VAPS system is a tool for designing, testing, and implementing user interfaces. A prototype or "mod- 
25 ded* system is the result of the desig n work done with design tools constituting the VAPS si mulaticn system . 
A user interface is 8 part of hardware and software that provides for human control and feedback. For a soft- 
ware program, human control and feedback can include graphs, menus and prompts. For a product such as 
a radio, the interface can include scales, dials, and knobs. Thus, the user interface is a bridge for a human to 
access an application. 

so The resultant virtualized "prototype "system is a graphics simulation of a control and display mechanism 

equivalent In appearance, form, functionality, behavior, animation, etc. to the modeled system. The virtualized 
prototype representation is also applicable to the generation of source doe implementing equivalentf unction- 
ality on a target system. Thus, as used herein, the prototype system is functionally equivalent to a modeled 
system. However, while the virtualized system is termed to be a prototype, i is not limited to in itial system 

35 development but may constitute the final target system. 

The VAPS system supports development and testing of dynamic, graphical front ends for existing appli- 
cations and maintenance of the user Interface by allowing changes to be madB visually. VAPS contains four 
primary modules or subsystems: (i) an Object Editor, (ii) an Integration Editor, (iii) a Logic Editor, and (iv) a 
Runtime Environment. 

to Briefly, the Object Editor is used by the operator to build the visual interface tothe application. Using Ihe 
Object Editor, the user draws objects that make up the graphical interface for the prototype. The objects are 
then assigned runtime behavior. The user employs the Integration Editorto define and build data connections 
or routes between the visual interface and an Applicat Ion program data. That is, af terthe display image is pro- 
duced using the Object Editor, the Integration Editor connects objects to each other, to data sinks, and to data 

« sources. At the system level, data connections can be made by defining global prototype/applications system 
variables. 

The Logic Editor allows the user to define and specify how events are to be managed. Using the Logic 
Editor, the user can develop a logic control program to manage events from the user interface. Finally, the 
Runtime Environment animates the prototype and allows the user to interact with the prototype, 
so In the VAPS system, the Object Editor is a window-based, graphical system used to draw and describe 
objectsthat makeup a man-machine interface (MM I.) The VAPS M Mis are collect ions of objects having a visual 
representation created by the user and afunctional behavbr assigned to the graphical representation. 

The Object Edtor allows the user to build the "functional behavior" exhibited by complex objects or dis- 
plays in runtime by specifying a list of object descriptions. The functional behavior of an object refers to how 
56 the graphic representation of an object responds to data received and generated by the prototype. 

The object descriptions define two types of objecta- input and output objects. In general, inputobjects react 
to user control to supply data to the simulation program representing the "prototype" or "modeled" system. 
Output objects react to data generated by the Input objects, as well as to internal and external data sources 
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of data. Output objects reflect the data by changing appearance in accordance with the predefined functional 
behavior of the respective object type. Thus, an output object such as a virtual meter might be graphically rep- 
resented by a. graduated scale and a pointer, the position of the pointer relative tothe scale being responsive 
to predefined data generated by the prototype system. 

s Input objects accept user inputs and pass data to the system for processing. Typical classes of virtual input 

objects include buttons, locators, menus, knobs, switches, event labels, input fields, etc. Virtual input objects 
can be defined to be responsive 1o a corresponding physical input device, such as a mouse, keyboard, touch 
screen valuator, or a custom device. 

Once the application program has processed the information, the reeultsaredisptayad on a videomonitor 

10 screen using output objects. Typical classes of output objects include dials, lights, plots, cursers. bar charts, 
tapes, output fields, etc. Thus, output objects are different representations enabling viewing of data generated 
by the application. When the data changes, the output objects are dynamically modified to reflect the new 

The functionality, i.e. behavior, of objects is defined using property sheets. The contents of a property 

15 sheet are dependent on the class of the object described by the property sheet For example, a property sheet 
for a dial dass {i.e., rotational) object defines the name of the object, display position origin, moving part, center 
of rotation, position pointer, initial pointer position, direction of motion, type of motion, and minimum and max- 
imum positions and corresponding values. 

As described, the Object Editor is used to draw the visual portion of the simulated prototype and define 

20 the functionality of objects. The Integration Editor Subsystem animates the objects by providing a data con- 
nection to the application data. Connections with the Integration Editor are made by specifying a correspon- 
dence or mapping between an object and a data source or data sink. 

For example, if a "Dial" type of output object represents the temperature in a boiler, the temperature can 
be changed by an external source. The correspondence between the temperature and the Dial reading is made 

is in the Integration Editor. Similarly, an input object that is created and functionally defined in the Object Editor 
is connected via globally defined data to provide user input to the application program. 

Integration produces a data connection allowing virtual input objects to transmit data and output objects 
to receive data. The data controls the functional behavior of a virtual object or alters the attributes of a virtual 
object, such as a displayed position or shape of the virtual object. 

30 Using the Integration Editor, data connections to and from a vrtual object are made through a data chan- 

nel. Uata connections between virtual objects can be made d redly from one virtual object to a series of other 
virtual objects. This is accomplished by storing the variable information from an input virtual object in a mem- 
ory location, referred to as a data channel, and connecting the one or more output virtual objects to the same 
memory location to read the updated variable. 

36 Data generated by data generators of the prototype interface are connected through direct point-and-click 
connections. Thus, an output virtual objects can directly receive data generated by the MMI. As used herein, 
"point-and-click" and "dick on" refer to designation of an element or virtual object dispayed on a video monitor. 
This is typically accomplished using a mouse, joystick, touch screen or other graphic input device to position 
a cursor displayed on the monitor (i.e., point) and activating a momentary switch to designate the element or 

■w virtual object at the cursor position (i.e., click or dick-on) for further processing. 

External user-defined application programs can also provide data. The memory locations for input and 
output virtual objects can be configured to transfer date to and from an external user application program, 
with the appBcation program reading and writing to predefined data channels. 

Finally, real devices can be directly integrated into the prototype with the interface. Such real devices Sr> 

45 dude input devices such as sensors, joysticks, trackballs, and keyboards. 

The Integration Editor maps user input and output information to data locations without regard to where 
the data originates. A pictorial interface maps data to provide a direct outlet to a virtual object called a plug. 
All vi-tual objects have such an associated plug, i.e., a labeled data location accessible under program control, 
to make the data available to virtual objects and applications programs. In comparison with having to program 

50 memory connections through a programming language, mapping using the Integration Editor allows tha ueer 
to define how the virtual objects of the prototype are to communicate with each other by passing data back 
and forth using plugs. 

Each mapping variable is defined by making a pictorial connection between the plug and the data source 
or sink associated with a virtual object Therefore, when a virtual input objects of a prototype are connected 
SS to other date sources or sirics, only the information mapping needs to be changed, not the individual virtual 
object definition. 

Collectors are used to define required combinational Boolean or mathematical functions for application 
programs requiring the combination of several input sources to drive a single output virtual object Collectors 
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can also be used to determine a critical stats by comparing an input value with a threshold value. 

Data connections are also used to interlace input objects of a prototype with an external application pro- 
gram. For example, interactive application programs may require control and display data interfaces to a pro- 
totype, The interlace provides data communications between virtual objects and externally provided data. Tne 
5 prototyping developmentsystem also defines and provides communications with databases using connecting 
variables and remote or local communications. 

The Logic Editor is used to model the prototype system by defining prototype system states and events. 
Trie system uses a Finite State Machine (FSM)mm»ptastlTeframeworkforMMIdedsiori-logic development 
This allows the prototype system to manage events in a context that takes system history into account and 
10 supports a fine-grained modelling system functionality, thereby enhancing system reliability. 

Prototype system behavioral modelling with the FSM concept produces a model containing a finite number 
of states. Each prototype state is entered and exited based on the occurrence of a virtual event Events can 
be user or application generated. 

To handle FSM modelling, the Logic Editor uses art Augmented Transition Network (ATM) language. The 
15 ATN language provides a structure forrecognlzing events and servicing transitions between prototype system 
states. Transitions are serviced by invoking routines and then transitioning or moving to the next state. 

The Logic Editor is used to produce an ATN program and define prototype system operation. The ATN lan- 
guage has two basic const ructa, the STATE and the EVENT. These constructs map to the nodes and links 
of a conventional slate diagram to define prototype Interface decisions, i.e., provide logic control and action 
20 routines. The general structure of an ATN program is as follows: 

STATE start_state 

{ initialize the state's parameters if 
f ( EVENT eventl) (condition) { response 

-> NEXTSTATE 
j EVENT event2) (condition) { response 
M -> NEXT_STATB 

etc. 

When a virtual eventoccurs, the protot ype system is in a particular predefined state. The prototype system 
checks that the event that has occurred matches one of the transitions (Le., an event and a condition) declared 
as in that state. If the event is matched and the condition is true, i.e.. valid and passed, the prototype system 
executes the response by executing a number of routines. The prototype system then proceeds to the next 
state, awaiting other virtual events. Thus, the ATN code Is used to define decision making of the prototype 
system. 

Input generated events can be received by the ATN program and output virtual objects cart be addressed 

40 by action routines invoked by the ATN program. The MM! can respond to vrtual and real events originating 
from many sources, inducting data entry, selection, discrete device input (such as a button, switch, input field, 
etc.), a continuous discrete device input reaching an increment (such as a potentiometer, knob or locator), time- 
out, periodic event, variable value crossing a threshold, voice input Dr a user-definable event 

In orderto service events, action routines are dispatched, i.e., invoked to service the event. The prototype 

« development system makes libraries of action routines available to fulfill a variety of functions including per- 
forml ng specif ic object operations, manipu hating graphical i mages and virtual objects on a virtual CRT screen, 
altering line types, patterns, fonts, and colors under program control, performing geometric transformations 
under program control, writing to output fields and reading from input fields, type checking for strings, com- 
municating with an external simulation via Ethernet or other specialized buses, and remapping object plugs. 

so Once the functionality of the user interface is completed using the Object Editor, Integration Edttor, and 

Logic Editor, the dynamics of the prototype are displayed at runtime in the Runtime Environment and thB user 
is allowed to interact with the prototype through various real and simulated input devices. The Runtime En- 
vironments permits a user to evaluate the prototype and operation of the final MMI. 

Using such a prototype development system, an essential part in the programming t he human-computer 

65 interfaces for the prototype is to specify the behavior of the interface to virtual events. Such events may be 
generated by input devices (buttons, locators and triggers, keyboards, selects, Function keys, touch screens, 
pictorial touch panels, voice interfaces) as well as by a timeout, a message from another system or a variable 
crossing a threshold value. 



required } 
> 

} 
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It is convenient to use a Finite State Machine (FSM) paradigm to respond to virtual events. Use of a FSM 
is suited to addressing sequencing problems often fou nd in operator i nterfaces, and allows structuring of the 
control logic in an orderly, regular fash Ion. 

When an FSM is used, virtual events are processed in the context of a history of prior events and system 
s statee. A combination lock provides a good example of events interpreted in a context of prior events. Only 
the correct sequence will open the lock. 

in another example, the pilot command "landing gear up" should only be serviced In the state "Welght- 
off-wheels" and not in the state "Weight-on-wheds". In a FSM, the system is deemed to be in one of a finite 
number of states. When input arrives, a transition is executed from the present state to the next state. The 
10 next state depends solely on the input and the present state. The output also depends solely on the input and 
the previous state. When applying trie FSM model to operator Interfaces, FSM input Is made up of discrete 
events and FSM output is the dispatch of a series of subroutines or setting of plug variables. If a logical con- 
dition is tested before undertaking the transiion, the FSM is an ATN. 

ATNs or FSMs can be programmed wth a text editor, by completing entries in a table or with a pictorial 
15 editor. However, textual completion of the table requires an h-depth knowledge of the syntax and structure 
of the ATN or FSM. 

Accordingly, a need exists for an improved method and structure for defining the operation of a simulation 
system without requiring a knowledge of programming, FSM or ATN languages. 

A further need exists for a method of, and structure for, defining a State Table using a graphical input and 
20 pull down, context sensitive menus. 

A need further exists for a system to visually relate input end output virtual objects using a graphic Input 
device. 

A need further exists for a system which presents a user selection of valid relationships between virtual 
objects and available states. 
a A need further exists for a tool to specify a desired behavior using angraphicinput device such as a mouse 
to point to, and designate, virtual objects whksh are port of the behavior and by designating desired virtual 
object behavior from menus. 

Summary of the Invention 

90 

The invention is directed to a context responsive menu driven prototype development and simulation sys- 
tem responsive to a graphic input device for defining relationships between and among prototype virtual ob- 
jects displayed on a graphics display such as a video monitor. The relationships aie defined by user completion 
of a State Tebte defining prototype system states, virtual events or triggers, virtual actions, and next states. 
35 Parameters are input to the table using a combination of images of objects appearing on the computer screen 
together with graphically designated menu selections and alphanumeric keyboard entries. Upon completion 
of the State Table, the parameters are translated into an Augmented Transition Network (ATN) or similar lan- 
guage for execution on a target platform running the simulation (i.e., workstation, personal computer, main 
frame, etc). Thus, the prototype development and simulation system and method according to the invention 
« directs the user to relevant valid choices defining relationships between virtual objects; displays and stores 
the relationships, actions, events, and responses as a spreadsheet-like State Table; and translBtes the State 
Table into a compilable program orexecutablelanguageto be run on a target platform (e.g. computer) system. 

According to the invention, prototype system performance is defined using a graphical input device to des- 
ignate virtual objects and define relationships between and among the selected virtual objects. State Tables 
w in the form of spreadsheets are completed by interacting with the image of en operator interface composed 
of virtual objects, and deriving virtual events or actiors by clicking on the appropriate virtual objects. 

Messages sent by a virtual object, or methods executed by it, can be arranged in a list (e.g.. in a pulldown 
menu) and Imported into the spreadsheet. For example, the event "Light XYZ is turned OFF" would be gen- 
erated by clicking on the Image of the object light XYZ. and extracting the event "Turn OFF" from a pop-up 
» menus. Actions impacting virtual objects are specified in a similar manner. 

In conjunction wit h the prior prototyping systems described, which provide a graphical design environment 
for virtual object appearance, virt ual object behavior and virtual object interfaces, the invent ion doses the loop 
for allowing full pictorial programming ii the domain of dynamic, graphical operator Interfaces. Pictorial pro- 
gramming allows computer users who do not have a traditional programming background to carry out tasks 
55 that normally requ ire programming knowledge, thereby expanding access to computer simulat ions. 

The invention is further directed to a visual method of, and system for, programming interact Ivb. graphical 
applications on a computer system which includes a graphical display and input devices, such as a mouse and 
a keyboard. The method is particularly well suited to programming graphical, dynamic, interactive operator 
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interfaces. Dynamic Operator Interfaces Include applications with a graphical output which changes to reflect 

t ha values of data represented by virtual objects. 

The visual application is made up of virtual objects, which have both a graphical representation and a be- 
havioral model. Thegraphical representation of a virtual object is defines using a graphics editor, or otherwise 
s imported into the prototype system. A behavioral model ia specif ied by a Property Sheetthat maps a behavioral 

model for a virtual object to the visual representation. The mapping is specified by pointing to features of the 

virtual object, such as line of motion or center of rotation, or typing-ln values. 

The behavioral model Is used to anlmale the virtual object hi a manner consistent wit h the class of object. 

For example, a "Dial" type of virtual object me be predefined to rotate while a "Scale" type virtual object trans- 
10 lates, and a mult i-state "Light" type virtual object assumes one of several graphical representations to simulate 

a physical light. 

Virtual objects are connected to, or associated with, data values so as to animate the objects in response 
to the data. As the data changes, the virtual objects are redrawn to reflect the changed data value. This data 
can be generated by physical input devices, other virtual objects, software programs residing on the same or 
15 another computer, etc. 

The subject computer prototype development and simulation system allows the user to specify the reac- 
tion lo events in a pictorial manner, in a visual object environment. For this purpose, a spreadsheet like State 
Table and the visual object collection coincide on the same graphical display. The State Table is filled in by 
pointing to lists of virtual events or actions associated wih the different virtual objects. The contents of these 
20 lists are dependent on the virtual object class. Event or action descriptions are entered into the respective 
cells of the State Table, in the form of descriptive strings of text This text describes the events or actions, 
and the event sources, or action destinations. 

The State Table allows the user to specify prototype state names, logical conditions, and the name of the 
next prototype state for processing the next virtual event, thus Implementing an ATN control paradigm of the 
25 user interface. This information is sup pi ied by a user employing a n alpha n umerie keyboard, selecting f rem lists, 
or copying parameters from other cells of the spreadsheet State Table. Other control paradigms can be im- 
plemented using the same method, such as FSM, Rule based systems, or Petri Nets. The data stored in the 
State Table is used to automatically derive the control program for the prototype. 

Furthermore. In conjunction with the Information stored in the virtual objects, a complete source code 
so program can be generated. The source code program, when compiled, linked, and executed in the presence 
of appropriate data, produces the same visual appearance, animation behavior and interaction behavior to oc- 
cur on a host computer system as on the original prototype development platform. 

The invention includes a State Table Editor as an interactive module forspecifying the behavior of a pro- 
totype using, for example, the commercially available Virtual Applications Prototyping System (VAPS). The 
as State Table Editor or definition module Interface Is aim i lar to a spreadsheet paradigm, in thai the behavior of 
a prototype. i.e., responses of virtual objects to data, events, and other virtual objects, is defined by completing 
a two-dimensional table, using a modern point, click and type interface. 

The State Table Editor replaces the Logic Edftor of VAPS to define relationships between and among vir- 
tual objects, events, states and ections. The State Table Editor Is compatible with various simulation and pro- 
40 totyping systems. Thus, Augmented Transition Network (ATN) language programs previously written using a 
Logic Editor are usabla in the subject prototype development and simulation system. In addition, ATN programs 
can be converted Into State Tables and State Tables can be converted into ATN programs. 

The State Table Editorallows the user to specify the behavior of a simulator prototype using aspreadsheet 
like interface. This spreadsheet it used to specify the desired behavior in terms of behavioral rules. Each be- 
« havioral rule iscaled a "Reaction". Areaction specif ies howthe prototype should respond to a specified event. 
In Its simplest form, a State Table specification for a prototype consists of a collection of Reaction Rules. 
This collection of rules is called a Reaction Table. More sophisticated prototypes can contain more than one 
rule set Each rule set is called a state. 

The State Table Editor has a "Point and Click' interface allowing the user to complete the state Table by 
so clicking on the prototype dements, le.. designating a virtual object by manipulation of a graphics cursor using 
a graphic input device such as a mouse. Then , using the mouse and/or a data entry device such as a keyboard, 
the user fills in the State Table to specify (he desired behavior of the virtual object 

The State Table Editor package includes context sensitive menus and windows 10 enable an inexperienced 
user to define prototype behavior. It allows the expert user to describe even complex behavior, by providing 
55 access to an entire computer programming language such as "C" or ADA. 

For example, if the user is specifying the behavior of a virtual object, then the State Table editor displays 
or "pops-up" a window which enumerates all the functions which can be performed by that virtual object 
If the virtual object is an input object, then all the input events which that virtual object can produce is 
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shown in a menu. If the virtual object is an output object then all the functions which can control the virtual 
object is displayed. Once the appropriate event or f unct on is chosen , a second pop-up Window altows the user 
to fill in the details/parameters of the virtual object. Default values are nominally provided. If the parameter 
can have only a predefined set of allowable values, then these values are also displayed in a pop-up menu. 
s The State Tab le Editor converts the sequence of key clicks and typed values into the appropriate textual 

syntax The textual syntax describing an event is the ATN syntax for finite state machines (FSM). Actions are 
described in a programming language such as "C\ 

The State Table Editor provides a front end to the ATN capability of the prototype development system. 
The State Table defined by the user la translated into an ATN program which is compiled and used to drive 
10 the target prototype system at runlime. 

The State Table Editor allows a user who has little or no programming experience to specify the desired 
behavior of a prototype by U3lng a moose to point to the virtual objects which are part of the behavior, and by 
choosing the desired behavior from context sensitive menus. 

The foregoing and other objects, features , aspects and advantages of the invention will become more ac- 
ts parent from the following detailed description of the present invention when taken ri conjunction with the ac- 
companying drawings. 

Brief Description of Drawings 

20 Figure 1 is a block diagram of a processor platform according to the invention; 
Figure 2 is a blank state transition table; 
Figure 3 is a display used to define a react ion rule; 

Figure 4 is a display used to define a reaction table for an input virtual object 
Figure 5 is a State Table implementing the reactions rules specified in the State Table of figure 4; 
25 Figure B is a State Table defining the operation of the depicted vfrtual objects; 
Figure 7 i3 a chart of reaction trigger sources; 

Figure 8 is a State Table defining using a FIRSTTIME trigger to define prototype responses; 
Figure 9 is a display depicting a Slate Table using an INITIALLY trigger to define prototype responses and 
a simulation virtual object; 
so Figure 1 0 is a State Table including an IMMEDIATE event; 

Figure 11 is a prototype modified to include two virtual control knobs; 

Figure 12 is a f taw chart depicting four phases of using the State Table Editor according to the invention; 

Figure 13 is a screen display according to the invention; 

Figure 1 4 depicts the display format of a Parameters Property Sheet; 
ss Figure 1 5 depicts the display format of another Parameters Property Sheet; 

Figure 1 6 depicts the display format of Parameter Value Windows: 

Figure 1 7 is a flow chart of State Table Cell selection according to t lie Invention; 

Figures 18-20 are flow charts of State Table Cell configuration performed according to the invention; 

Figure 21 tea table or EVENTS for two example classes of objects. 
40 Figure 22 is a table of EVENTS not requiring specification of an object 

Figure 23 is a table or ACTIONS for two example classes of objects. 

Figure 24 is a table of ACTIONS not requiring specification of an object 

Figure 25 is a flow chart of system operation in a relevant Choice mode of operation; 

Figure 26 is a flow chart depicting determination of default parameter values according to the invention; 
« Figure 27 is a flow chart depicting system operation In response to a user parameter selection; 

Figure 26 is a flow chart depicting system operation in response to a user parameter value assignment 

selection; 

Figure 29 is a graphic image of a simulation display; and 
Figure 30 is s completed state transition tablB. 

50 

Best Mode for Carrying out the Invention 

A computer platform for the development and simulation of a prototype is depicted in Figure 1. Acentral 
Processor 1 02 is connected to a Math Coprocessor 104 for executing arithmetic and logical operationsrespon- 
ss sive to a program of instructions stored in Random Access Memory 106. The program of instructions includes 
an operating system such as Unix, software components of the prototype development system including (I) 
an Object Editor, (ii) an integration Editor, (iii) a State Table Editor, and (rv) a Runtime Environment module, 
and date generated by each to define the prototype. The Runtime Environment module also supports execu- 
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tion of a simulation of the prototype in response to prototype behavior specified by a State Table defined by 
the State Table Editor. 

To speed memory access, the platform includes a Cache Memory Controller 1 08 and Cache Memory 110 
for storing frequently used instructions and data. Mass system storage is provided by Disk Storage 112 and 

s Disk Controller 114. Central Processor 102 can access Disk Storage 112 through Disk Controller 114 either 
via Input/Output Processor 116 or over Data/Instruction Bub 118 under the control of Bus ControUer 120. In- 
put/Output Processor 116 also receives and formats data from Keyboard 122 and Graphics Input Device 124 
via Graphics Input Interface 126, and Touch Panel 126. Graphics Input Device 124 may include a mouse, joy- 
stick, light pen, digitizing tablet, trackball, keypad, or other device for positioning a video cursor. 

» The resultant prototype simulation is generated by Graphics Processor 1 30 under the control of Central 
Processor 102. Graphics Processor 130 generates alp ha numeric and primitive graphic Image data eltherunder 
direct control of Central Processor 102 or in response to lists of instructions or vectors stored in Random Ac- 
cess Memory 106. The image data is stored as a bit mapped image in Image Memory 132 and displayed by 
Video Monitor 134. 

« The computer platform also Includes an External System Interface 136 for exchanging data with another 
computer system. Speech recognition is supported by Speech Recognition Processor 138 which receives 
speech sounds at microphone 140. interprets the sounds and provides a corresponding ASCII output string 
to (nptrtJOutput Processor 116. Finally, synchronous type operations of the computer platform are synchron- 
ized by System Clock 142 which also provides periodic interrupts signals at predetermined intervals set under 

20 software control. 

Referring to F gure 2, the State Table sa two dimensional table withfour columns and an unlimited number 
of rows. The four columns are: 

1- Stale 

2- Event or Trigger 
25 3 -Action 

4 - Next State 

A State Table represents the behavior of a prototype simulation as a collection of Reaction Tables. The 
present invention provides a method of, and apparatus for, filling In the state table of Figure 2 by controlling 
virtual objects on Video Monitor 134 in response to user input signals derived from any or all of Keyboard 122, 
so Graphic In put Device 124 or Touch Panel 128. Context driven windows are displayed on Video Monitor 1 34 in 
response to user designation of cells of tha state table to be completed, the windows providing lists of valid 
table entries for the selected cells. 

A reaction rule specifies how the prototype should respond to events. The Prototype must respond to a 
user operating real input devices in the which affect virtual input objects of the prototype. Using Keyboard 
35 1 22. Graphics Input Device 1 24 and Touch Panel 12a the user may push a virtual button, rotate a virtual knob 
or slide a virtual potentiometer graphically represented on Video Monitor 1 34. The simulator system responds 
to the inputs by affecting the oulput virtual objects of the prototype as displayed on Video Monitor 1 34. 

For example, referring to Fig. 3, in response to the user activating a joystick or mouse of Graphic Input 
Device 124 or by manipulating the corresponding object using a touch screen to cause simulated rotation of 
40 the illustrated virtual knob the simulation system adjusts the illustrated virtual speed dial to display the value 
of the knob. This is described In a reaction rule as follows. 

•KNOB speed_knob- 
which is called the event part of the reaction rule, while 
"Drive_Diair. "speed_dial", KNOB_VAL);" 
43 Is called the action part of the reaction rule. The even! or trigger part of the reaction rule is written using a 
syntax which is unique to the simulation system (i.e., ATN), white the action part is written in the *C" program- 
ming langauge. 

More complex reaction rules are possible. For example, referring to Figure 4, the prototype may contain 
a virtud knob and a virtual light selectively displaying one of two Illumination levels, e.g., black and white. In 
so the example, the light t urtv3 white whenever the value of the knob is less than 500. and turns black when the 
value of the knob Is greater than or equal to 500. 

This relationship can be expressed with a pair of reaction rules. A collection of one or more reaction rules 
is called a reaction table as depicted in Figure 5. The reaction table depicted in the figure has two reaction 
rules, with the event part of both reaction rules being two lines long. The first line gives the primary event or 
ss trigger which must occur for this rule to be triggered. This primary event is the knob turning. The second line 
of this reaction rule is called Hie condition of this event The condition specifies that this reaction rule should 
onty be taken if the value of the knob is < 500. Conditions are also written in the "C* programming language. 

Each of the two reaction rules have the same primary event as triggers. The rules differ only in thecon- 
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dition which must be true for the reaction to take place. 

In t he prototype development system . a user enters a reaction table in t he form of a state tab) e. A prototype 
development State Table consists of one or more react ion tables . On ly one react ion table is active at any given 
time. A. simple si mutator system State Table consisting of only one reaction table is depicted in Figure 5. 
s Since the State Table has only one reaction tat**, the State and Next State columns of the Stats Table 

are left Wank. When a State Table has more than one reaction table in it. each reaction table must be givei a 
name. The name of each reaction table Is called a state, When a State Table is executing, only one reaction 
table within the State Table Is active at any given time. 

A Slate Table thus consists of one or more states. When a prototype whose behavior is specified using a 
10 State Table is running, the prototype simulation system wil be in one of the states given in the State Table. 
Thus, the word state is synonymous with "mode". 

A prototype may be controlled by more than one State Table. For example, different virtual objects may 
be controlled by separate State Tables. 

Each reaction rule can specify that a change in state should occur as part of the reaction to the rule being 
15 triggered. Thischange in state occurs in response to an Event and satisfaction of the corresponding Condition. 
When the event occurs, the prototype executes any actions specified In the action column in the ru e. After 
the action has executed, the state of the State Table changes. Once the state of the State Table has changed, 
only the reaction rules specified for that nsw state are active. 

Figure 6 depicts a simple prototype and State Table having two states. The two states are called LOW 
20 and HIGH. Whan the State Table is In the LOW state, the value of the knob is copied to the dial. When the 
State Table Is In the HIGH state, the value of the knob is doubled and copied into the dial. Flipping the GEAR 
switch toggle* the State Table from the LOW to the HIGH state and back again. 

Each of the two reaction tables shown in Figure 6 has two reaction rules. The first reaction rule in the 
LOW state specifies an action in response to the user turning the knob. Since turning the knob does not 
2S change the basic mode of the Prototype, no mode change is required. Therefore, the NEXT STATE column is 
left empty. 

The second reaction rule is also part of the LOW state. It specifies t hat whenever the switch is set to HIGH 
the State Table switches to the HIGH state. The dial Is also reset to twice the value of the knob. 

The HIGH state is structured similarly to the LOW state, except that, in the HIGH state, twice the value 
so of the knob is sent to, and displayed on, the dial. As also defined in the HIGH state, the prototype simulation 
reverts back to the LOW state when the switch is switched back to LOW. 
Another term used to refer to reaction rules is transitions. 

The reaction rules described above all had events which were triggered by Virtual Input Objects. Other 
reaction triggers are possible. Reaction triggers can have one of the following sources: 

1 - An input virtual object generating events; 

2 - Clicking the mouse on the screen in an area unoccupied by a virtual object; 

3 - Events generated by Computer Programs; 

4 - Input from a Serial device of communications ilne; 

5 - Time based events; 

io 6 - Variables crossing thresholds: and 

7 - Inputs from real devices which are mapped to virtual objects. 

Other reaction tr ggers are s hown in the table of Figure 7. The first group of events are examples of object 
events, i.e. events which are generated by using the mouse with virtual objects In the prototype. Some virtual 
objects have more than one input event associated therewith. Locators (e.g. . mouse or joystick) have three 

45 different input events. The LOCATOR_DOWN and the LOCATOR_RE LEASE event occurupon pressing and 
releasing the mouse in the active area of the Locator. The LOCATOR_DRAG event occurs when t he location 
of the Locator is changed using the graphic input device on the active "touch" area of th B display screen within 
the defined area of motion for the Locator. 

Field events occur when the user types data into an input field, for which a "Read „Fleld{)" f unction has 

so been Invoked. A FIELD VALID eventoccurs if the value typed in is valid with respect to the syntax for the field. 
AFIELD INVALID eventoccurs If the value entered is not valid. In either situation, a FIELD event (without t he 
VALID or INVALID qualification) is generated. 

Referring to the Other Mouse Input Events entry, a NOD EVICE event occurs if the user clicks somewhere 
in the prototype that is not within an active area of a virtual object This event is received only if it is enabled 

ss in the Runtime Environment using the "Enable No Device Event" field in the Operational Settings of the 
Configuration menu. 

The ANY EVENT t rigger, responds to any event that occurs from input virtual objects, as well as any PER- 
IODIC event which occurs. This is useful to detect and I09 stray events. 
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The next f ivG events are generated internally within FSM in response tostate transitions. The FIRST_TIME 
event occurs the first time that a given state Is entered. This event does not occur a second time if the state 
is entered and exited and entered again. The FIRSTJTIME event Is often used to perform initial loading of a 
frame, wherein a frame is a collection of objects. 

5 The INITIALLY event occurs each time that the a new state is entered from another state. This event is 

used to configure the display for the newly activated state. This may include the loading frames or changing 
the values of variables used by other transttlons within t fiat state. In contrast, the F1RST_TIME event is used 
to load the frame with the virtual objects only the first time the Stats Table is initially entered. Thus, the next 
transition has an INITIALLY trigger. Inthe previous example, wheneverthis transition is entered from another 

10 state, the dial is reset to the value of the knob. The dial is also set to the value of the knob each time a new 
input value is received from the knob. Whan the user toggles the gear switch to HIGH, the mode is switched 
to HIGH. There is no action associated with this state. Instead, the system has only associated a change in 
state with this reaction rule. Once in the HIGH mode, the INITIALLY reaction is performed. This causes the 
dial to be set to twice the value of the knob. 

15 The use of the IMT1ALLY event helps make the State Table clearer and easier to understand. Compare 
the State Table shown In Figure- 8 with that shown in Figure 9. In the State Table of Figure 8, it was necessary 
to place an action in the transition to cause the state to change when switching from one mode to another. 
Although this resulted in the desired behavior, it is conceptually incorrect since all the uses of the action rou- 
tine to set the value of the dial to twice the value of the knob should be encapsulated within the HIGH state. 

20 The IMMEDIATE event is invoked each time that a state is entered, as well as after each reaction in which 

the state did not change. An application of the IMMEDIATE event is to perform date logging. 

Referring to Figure 10. different transitions are taken in responseto different input events. In all cases of 
a transition being taken, a log entry is made. Another use of the IMMEDIATE event Is when there is an output 
virtual object in the display which can be affected by one of many input virtual objects. In these cases, rather 

is than driving the output virtual objectf rom within, transition routines can be factored out which drive the output 
virtual object, and place it in the IMMEDIATE action. An example of such a use of this event is depicted in 
Figure 11. 

Referring to Figure 11. the prototype includes two knobs. The prototype displays the sum of the values 
represented by the two kinds on the dial. This is accomplished by storing the value of the knob in the reactions 
50 which respond to the knobs being turned, and driving the dial with the sum of the values In the IMMEDIATE 
event response. 

The FINAL event occurs when leaving a state. It occurs only if there is a NEXT_STATE filled in, and the 
NEXT_STATE Is different from the current state. In this case, the action associated with the FINAL event is 
executed before the transition to the next state. Atypical usb of the FINAL state is to undo any actions per 
35 formed in response to the INITIALLY event upon entering the state. A typical example is unloading of frames 
which were relevant only within that state being left. 

The Pseudo Event COMMENT, has no operational effect. The remainder of the tine is treated as a com- 
ment and ignored. Comments can also be entered as "C" language comments, i.e. between /* and */. For ex 
ample: 

40 t* This is a comment */ 

The prototype design system also provides two time based events. The first one is called PERIODIC. The 
PERIODIC event occurs repeatedly at the specified frequency. The syntax for this is 
PERIODIC nn HZ 

where nn is the frequency of the event in HERTZ. The PERIODIC event can be used to Increment counters, 
« complement samplers which change the display at specified intervals. 

The clock which generates the PERIODIC event is a global dock. This means that the PERIODIC clock 
is not reinitialized upon entering a state. 

The other time based event is the ON TIMEOUT event. The syntax for this is: 
ON TIMEOUT nn SECS 

so h this notation, nn must be a positive integer. Thia event occurs nn seconds Bfter entry of the state and is 
usBd, for example, to detect if the user has tailed to timely enter data or to determine If the user has finished 
entering data. The ON_TtMEOUT is reset upon occurrence of a reaction or state change. 

In addition, there are two user defined everts. The first user defined event is called USER which is gen- 
erated by a user program. The USER event takes two parameters; the name of the USER event and the value 
S5 of the event. 

The syntax for using USER generated events is: 
USER UMr_Even1„Naittt Usar_Event_ValiMf. 
The USER event is generated within another ATN using the routine Generate_User_Event The syntax is: 
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Gonar»te_U*Br_Ev«nt("UsBr_Ev«n^N«nw"."UB»r_Ev«iMtoliiol; 

User generated events are used where multiple State Tables are concurrently executing. These State Ta- 
bles can signal each other using customized events. 

As described above, the prototype development system consists of four main components. The Object 
s Editor (OE) allows the user to draw virtual objects by specifying the shape, color, placement and texture of 
the graphical objects and ascribe to each an appropriate class of behavior. For example, if a dial Is drawn, It 
can have arty shape, or color. 

The Integration Editor (IE) allows the user to specify the names of data variables which control the shape 
of virtual objects. For example, if a virtual object is a dial, a particular variable can be madB to control the 
10 angle of the "needle/pointer" on the dial. 

The third component is the State Table Editor. This component allows the user to specify more complex 
behavior than possible using the Integration Editor of the previous systems. For example, in the Logic Editor 
(LE), the user can arrange to have virtual objects appear or disappear when a particular button is pressed 
and a particular condition is outstanding. The State Table Editor instead uses State Transit ion Programs, aug- 
« mented with "C" programming code. The State Table Editor replaces the LE of previous simulator systems. 

The fundamental difference between the State Table Editor and the LE Is ease of use. The State Table 
Editor significantly reduces the user generated and supplied programming code required to specify the be- 
havioral specification for a simulator prototype. 

The final component of the simulator is the Runtime Environment (RE). The RE integrates the information 
to entered using OE. IE and State Table Editor and runs the specified interactive graphical display. 

The are four phases to using t he State Table Editor are shown in t he flow chart of Figure 1 . When invoking 
the State Table Editor, the user initially sets up the environment of the prototype development system. Once 
the user has set up the environment, the State Table Editor can be used. The prototype development system 
records how the user set up the environment the test time the State Table Editor was used. 
is The following is a description of setting up the environment and editing the State Table. 

The State Table Editor consists of the following components: 

1. The State Table Spreadsheet 

2. Displayed Collection of Graphical Objects called a Frame; 

3. Command Menu Bar 

30 4. Text Entry Window; and 

5. A Displayed List of Choices that are relevant to the current mode. 

A typical State Table Editor environment is shown in Figure 13. The State Table Includes a menu bar 30 
describing options available in the State Table Editor. A Frame Display 32 graphical y depicts virtual objects 
to be defined by the State Table Editor. Relevant Choices List 34 provides user selectable commands. Text 

35 Entry Window 36 slows the user to enter virtual object labels and ATN instructions. The resultant system def- 
initions are displayed in State Table 36. 

By editing a State Table, a user specifies the behavior of a particular graphical display, i.e., a Simulator 
Picture Frame. This is a frame created using the OE. This frame contains one or more graphic objects. The 
user selects one such frame to be controlled by the State Table using the State Table Editor. 

40 The user also designates whether a new State Table is to be created or if an existing State Table is to be 
edited, in the latter case, an existing State Table Is retrieved and loaded into memory for editing. During the 
editrig, the user may load a different State Table, or a different frame. The other components of the environ- 
ment are placed in the environment by the State Table Editor without user intervention. 

Loading aframeora preexisting State Table is accomplished by choosing the appropriate entry fromthe 

45 puJI down menus in the menu bar, and choosing the desired file name. 

Once the user has set up the State Table environment, the State Table can be edited. Editing the Stele 
Table includes filling in the cells of the State Table. Referring to Figure 13. the State Table is organized li ke a 
four column spreadsheet. The State Table can have as many rows as required to specify the desired behavior. 
Each cell in the State Table contains information specific to that call. 

» For example, if a particular cell in the even t column has aa is "subject" a particular knob, Ihen the relevant 
list for that cell includes the various events that can be generated by the knob. This may include the events 
"MOUSEJJP", *MOUSE_DOWN". 1CNOB_NEW_VALUE*, "KNOBJHASJWNVAU*. 'KNOB_HAS_MAX_VAL". 
The cells in the state column and the next state column contain the names of states as previously designated 
by the user. 

55 The relevant list for the state column is the list of all the statee In the Slate Table. This includes arty state 

which is already defined by another cell in the state or next state column. 

The relevant list for the next state column Includes all the states entered in the state table. In addition, 
certain keywords are also listed including "EXIT", "SUSPEND", "RESUME" and "RETURN*. 
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Both the calls in the Event column and the Action column can refer to a virtual object as their "subject". 
Relating a cell to an object is performed by clicking on an event or action cell, and then clicking on a virtual 
object in the frame. The relevant list tor a celi in the Event column depends on the virtual object referred to in 
the cel. 

If no virtual object is associated with a cell, then a list of events that do not require a virtual object is dis- 
played. This includes events such 8S "STATE TABLE_STARTED\ *STATE_ENTERED", and 
"STATE_EXITED". If a virtual object is associated with the cell, the relevant list then includes everts t hat can 
be generated by the virtual object For example, if a particular cell Inthe event column has an associated virtual 
objBct comprising knob, then the relevant list far ihal cell includes the various events that can be generated 
by the knob. This may include the events , MOUSE_UP', "MOUSE_DOWN\ "KNOB_NEW_VALUE". 
"KNOB_HAS_M IN_VAL", "KNOB_HAS_MAX_VAL". 

The relevant list for a cell in the Action column depends on the virtual object which is referred to in the 
cell. If no virtual object ia associated with the cell, then actions thatdonol require a virtual object are included 
in the relevant 1st. This includes actions such as "CHANGE_STATE TABLE_PR10RITY", 
■START.COMMUNICATIONS*. and TERMI NATE_PROGRAM". 

If a virtual object Is associated with the cell, then the relevant list comprises a list of actions which can bs 
performed on the virtual object For example, if a particular celi in the action column has as its associated 
v rtual objects particular multi-stale light virtual object, then the relevant list for that eel includes various ac- 
tions that can be performed on the specified virtual object type. This may include the actions 
"CHANGE_STATE", "CHANGE_TO_DEFAULT_STATE", "HIDE", "REVEAL", "BLINK*. 

The list of events and actions which are associated with each object type is defined by a user alterable 
dictionary including a list of events and actions supported by each object type. This allows the set of virtual 
object classes to grow, or the list of events and actions which virtual objects support to change, without chang- 
ing the state table program source code. 

Once an event or action is chosen, the event may be assigned one or more parameters. If a parameter is 
to be assigned, an additional window pops up ard to allow the user to enter values for the parameters. Typical 
parameters property sheets are depicted in Figures U and 15. 

Each field in the property sheet has a "set of allowed values*. Some fields may receive numeric values, 
while other receive a name of a state of the virtual object, and others require the user to choose from a par- 
ticularllstof values. When the uaer clicks on a particular field in the property sheet, a parameter value window 
pops up, i.e., is displayed. This parameter value window allows the U3er to enter the value for the particular 
parameter. If the parameter has a fixed set of allowed values, a list of those values is shown. The user then 
clicks on I he desired value. 

The list of allowed valuea may come from one of two sources. It may come from the virtual object which 
is the subject of the cell being edited. For example, if the virtual object is a multi-stale light, tha names of the 
states of the light may be shown. Alternatively, the field may be defined as a "C" or other computer language 
type. If the field type is an enumerated type, i.e., limited to a predefined set of values, the list of allowed values 
is taken from the enumerated list for that type. For example, if the type of the field is ANCHORJYPE and 
the type was defined as: 



typedef enum 

{RE L AT I V E_TO_CRT , RELATIVE_TO_FRAME , ABSOLUTE, 
SCAL E D_TO_F RAME } 
ANCHOR_TYPE; 

then the appropriate values are used for that field. 

Finally, if thefield is numeric or an arbitrary string, then a text entry field is used. Examples of parameter 
value windows are depicted in Figure 16. 

The basic sequence of completing or filing in e State Table is to select a cell by clicking on it and either 
typing in a value using t he ed it text window or clicking on t he appropriate items from other windows a nd menus. 
This process is depicted in the flow diagram of Figure 17. 

Initially, the user selects a cell and the system configures the State Table Editor for the selected cell. De- 
tails of the system table configuration routine are shown in Figures 18-20. 

Referring to Figure 18, the State Table Editor first removes any preexisting windows from the display. If 
the selected cell is a State celt, then a list of state cell keywords and all known state names are displayed in 
a window. If the cell is a Next State cell, then a list of next state cell keywords and all known state names is 
displayed In a window area. If the cell is an Event cell, event cell processing is performed in accordance with 
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the flow diagram of Figure 19. 

Referring to Figure 19, if the cell has a subject, then the relevant choice list displayed includes a list Of 
events for that virtual object type and the specified cell subject If there is no subject associated with the Event 
cell, then a list of event oor responding to that type of cell is displayed. A timeout i3 an example of en event 
£ not requiring specification of a subject 

The list of relevant choices depends on the class of t he object the list being different for different object 
classes. Examples of lists of events related to specific object classes are depicted in Figure 21. One such list 
is needed for each object class. Similar classes of objects ma/ have some events of each list in common. The 
lists are stored In a data base, 
10 If the cell does not have a subject, then a list of events which require no subject is displayed. An example 

of such a list of eve its ts shown in Figure 22 

After the relevant choices associated with the Event cell are displayed, a check is performed to determine 
if a relevant choice forthe cell has already been selected. If so, thecurrent parameter values are set accordingly 
and displayed in an Event parameters window, 
w ifthe cell is notan Event cell, then the cell is Interpreted asanActfoncell. Referringta Figure 22, areievant 
choice window is displayed corresponding to whether a subject is associated with the Action cell. Action cells 
not requiring a subject include routines contained in a "C" library such as for computing a random number. The 
system further displays a Relevant Action parameters window if a ceB has a relevant choice already chosen. 
The list of relevant choice actions is determined in a way similar to that used to determined relevant 
20 choices of events. Example of lists of relevant actions for different object classes are depicted In Figure 23. 

tf an action cell does not have a subject, then a list of actions which require not subject is d isptayed, as 
shown by example in Figure 24. 

Referring again to Figure 1 7, once configured, the system waits for an input from a user. The input is clas- 
sified into six types: the user (i) types text into an edit window, (if) picks an edit command from a window, (iii) 
35 chooses a virtual object from a frame, <iv> selects an item from a relevant choice window, (v) selects a para- 
meter form a window, or (vi) selects a parameter value from a parameter value window. If the user enters text, 
the system wa ts until the user finishes typing and supplies the text to the prototype development system for 
parsing and further processing. 

If the user designates an Edit command, the State Table Editor performs the chosen editing function. Se- 
30 lection of a virtual object causes the Stats Table Editor to make the selected virtual object the subject of th B 
cell. 

If the user selects a relevant choice from a window, processing proceeds as shown in the flow ehartcf 
Figure 25. As depicted, the selected choice is assigned to the cell. Ifthe choice requires one or more parame- 
ters, processing is performed as shown in the flew chart of Figure 26 to determine an appropriate default value 
35 for each required parameter. 

Referring to Figure 26, a check is first performed to determine if the relevant choice provides a default 
value. If not, then, if t he type of choice is a characteristic of t he virt ual object which is the subject of t he current 
cell, then a default parameter is provided corresponding to the specification of the virtual object Alternatively, 
ifthe relevant choice is not a characteristic of the virtual object then a default value is supplied based on the 
« type of choice selected. 

Referring again to Figure 17, ifthe event or action window is up and the user selects a parameter from 
the parameter window, processing continues as 3hown in Figure 27 to allow the user to select a parameter to 
be specified. The chosen parameter is then made the current vaiue, Alternatively, if the parameter value list 
window is up and the user selects a parameter, processing continues to allow the user to choose a parameter 
49 value as depicted in Figure 28. 

Thefollowing description is an exam pie of using the State Table Editor and prototype development system 
according to the Invention. 

The user first enters the Object Editor by typing 'object_editor' from the command prompt or by clicking 
on a command icon at a computer work station. Then, the user defines or retrieves a previously stored protc- 
» type. In this example, the prototype is the simplified control panel 20 for a train a3 shown in Figure 29. 

As described and defined using the Object Editor and Integration Editor, the prototype has two output vir- 
tual objects and two input virtual objects. The two output virtual objects are traffic light 22 and speedometer 
24. The Input virtual objects are push button 26 and a gas pedal represented by potentiometer 28. 

For example, suppose that the operator determines that the control panel is to exhibit the following be- 
55 havior. The main control ie to be the speed control. i.e.. sliding potentiometer 28. The train can travel at any 
speed from 0 to 200 miles per hour. Control panel 20 includes traffic light 22 which informs the operator of 
the maximum allowed train speed. If traf f ic light 22 is green, then the train may proceed at any speed. However, 
as indicated by the eroeehatched area on the speedometer, speeds greater than 180 mph are dangerous and 
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should not be used except under special circumstances. 

When traffic light 22 is yellow, the train may not travel faster than 20 mph. When traffic light 22 is red. 
the train should stop. 

I n the prototype, (raff ic light 22 should bo controlled by push button 2$. Repeated pressing of push button 
5 26 causes the traffic; tight sequence through the colors red, green end yellow. Initially the traffic light is red. 
Pressing button 26 once makes it turn to green. Pressing it again makes the light change to yellow. Pressing 
it again makes the light change to red. 

After defining the desired operating rules, the user creates a State Table to implement these rules. The 
user creates an empty State Table using a main pulldown menu to select "State Table.* This causes a display 
10 of a pulldown menu from which the user selects the "New StatB Table." 

Next, the user brings up the Pick By Name window. Form the main pulldown menu, the user selects util- 
ities" and, from the resultant pulldown menu, selects "Pick By Name". Than, rrorn the State Table definition 
main pulldown menu, the user dicks on "Utilities". From the Utilities menu, the user clicks on "Focus Selec- 
tion". From Focus Selection, the user clicks on the second entry in the menu, connecting the Pick By Name 
is menu to the frame which the user has loaded Into a Show window for display. Then, the Focus Selection win- 
dow is removed by clicking on an appropriate field of the Focus Selection window. The user can then begin 
to enter the de9tred prototype behavior into the State Table. 

The user must first arrange to have the Frame loaded when the prototype begins operation. This Is ac- 
complished by clicking on the first cell in the column of the State Table Editor labelled EVENT. IN response, 
10 a windowtltled "Relevant Choices" Is displayed. From this window the user selects the entry "FIRST_TIME\ 
The selected cell in the State Table Editor now contains the text FIRSTJIME. 

The user next dicks on the first cell in the column labelled Action. In response, the "Relevant Choices" 
window changes to display context appropriate choices. In the "Pick By Name" Window, the user clicks on the 
top element which is the frame Itself. The "Relevant Choices" window now displays new choices applicable 
25 to the frame. 

The user next selects the entry "<3et_Fname" from the Pick By Name window. As a result, a window titled: 
"Arguments of Function: GetJFramofJ" appears. This window allows the user to change the parameters to 
the Get_Frame(J action. Since the default parameters are appropriate, no Changs is required. 

The~first state to ba created is titled RED. The user initially clicks on the first cell in the column labelled 
30 Next State, highlighting the display of the cell. As before, a Relevant Choices window will appear. The UBer 
clicks on the cell again causing a data entry wwidow to appear. This follows the general rule in the State Table 
Editor that a second click on a cell causes a data entry window to be displayed, allowing the user to manually 
input values using alphanumeric keyboard entry. 

The user then types "RED" into the table followed by a carriage return from the keyboard. The value RED 
36 then appears in the NEXT STATE Cell. Above the State Table editor work area are State Table edting com- 
mands. The user clicks on the command "Append Transition" to add a second empty row to the state Table. 

Next, the user clicks on the second cell in the column labelled State. From the Relevant Choices menu, 
the user selects "RED" to create the state RED. 

In the state RED, the user first sets the dial representing the spe edometer to 0. To do this, the user dicks 
40 on the second cell in the EVENT column. From the relevant choices menu, the user selects the INITIALLY 
item. The initial behavior of the system in response to entry into the RED state is then spedf ied in tho Action 
column. 

To specify the behavior, the userdicks on the second cell In the ACTION column. Fromthe Pick_By_Name 
window, the user clicks on and highlights DIAL. This causes the da) to be made the subject of the cell. 
45 Fromthe RelBvantChoices menu, the user next clicks on the Drive_Dial entry whereby window Arguments 

To Function: Drive_DiaJ are displayed. Since ell the default values are appropriate, no change is made. 

The user again dicks on the Action cell causing a data entry window to be displayed. The user adds a 
second Hank line and selects the traffic lightfrom the Pick By Name window. From the relevant choices menu, 
the user selects "Drive_Llghr.Slncathedefault value given is the RED mode of the light, no change is required. 
SO The user must next describe the behavior while in the RED State. The behavior in the RED state is to go 

to the GREEN state when push button 26 is operated or "dicked*. No response is required when the user 
changes the speed using the potentiometer in the RED state. 

To describe the behavior that should occur in response to button 2$ being clicked in the RED state, another 
transition must be added. The user first clicks on the command "Append Transition". Since this transition is 
es e continuation of the description of the behavior of the RED state, tho user need not fill In the STATE name 
column. 

The user then clicks on the EVENT column of the newtransition which is the third eel! inthe column. From 
the Pick By Name window, the userdicks on the "Desired Speed Button." Alternatively, the user can dfck o/i 
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the picture of the button, using the iaft button on the mouse. 

Next, from the Relevant Choices menu, the user picks the BUTTON selection resulting in display of default 
event parameters. Since, In the example, all the default values are correct, no cliange Is required. 

Having defined the RED state, the user next defines 1 he GREEN state. To do this, the usercfcks on the 
5 NEXTjSTATE cell of the transition. Clicking on the cell a second time causes a data entry window to be dis- 
played. The user defines the name of the state by typing "GREEN". 

Next, the user fills in the GREEN State parameters to define the behavtorforthe GREEN state. First, the 
user creates the GREEN state by clicking on the 'Append Transit ton" command. Since, three transitions are 
need In the GREEN state, the user clicks on the "Append Transition" command two more times, for a total of 
10 three times. 

The user next clicks on the STATE cell of the first of the three transitions that have just been added. From 
the Relevant Choices Menu, the user clicks on GREEN. Now, the user clicks on the EVENT cell of that tran- 
sition. From tha Relevant Choices menu, the uaar defines the initial stats by clicking on INITIALLY. 

Theirtitisl behavior in the GREEN state Is almost Identical to the required behavtorforthe INITIALLY event 
« of the Red state. Therefore, the quickest way to enter t his action is to copy it from the RED state, and then 
change the details. To do this, the user first dicks on the action tell of the INITIALLY transition in the RED 
state. The user then clicks on the COPY CELL command. Next, the user dicks on the action cell of the first 
transition of the state and then clicks on the PASTE CELL command. 

Next, the user clicks on the first row of the action cell causing a Function Parameters: Driv»_DialO win- 
20 dow to be displayed. The user clicks on the second field in this window. From the new pop-up window dis- 
played, the user selects the POTEN_VAL entry. 

The user now clicks on the second row of this cell The Function Parameters: Drlve_Lightfl window is 
displayed in response. The user clicks on the second field In this window. This brings up a window with a list 
of modes relevant to the light. In the present example, the light can be in the "RED", "GREEN" or "YELLOW 
is modes, these modes therefore being displayed in the window. The clicks on the "GREEN" entry thereby en- 
tering "GREEN" value as the Drlve_Light parameter. Alternatively , the user can dick twice on the second field. 
In this case, a data entry window wilt be displayed into which the user can type the value "GREEN". 

The next task in defining the operation of the prototype is to define the cor rect response to the usercllcklng 
on the button, or sliding the potentiometer. To do this, the user dicks on the second eel in the EVENT column 
so of the transition. From the Pick By Name window, the user clicks on the potentiometer Desired Speed. Now, 
from the Relevant Choices menu, the user dicks on POTEN. Next the user fills in the appropriate response 
to a change in value of the potentiometer. 

To define the appropriate response to adjustment of the potentiometer, the user clicks on the Action cell 
of the transition. From the Pick By Name window, the user clicks on Traffic_Light. From the relevant choices 
35 window the user dicks on Driw_Dial. The Function Parameters: Driv«_LlBhtQ window is then displayed. 
In this window the user clicks on the second cell which has displayed therein ZERO_VAL. A pop-up window 
displays available values. The user must then click on the POTEN_VAL value. 

To define the YELLOW State, the user first creates three transitions using the Append Transition com- 
mand. The first transition describes the initial behavior of this state. The user first clicks on the STATE column 
« of the first transition of the YELLOW state and YELLOW is selected from the relevant choices column. The 
user next clicks on the EVENT column, and selects INITIALLY from the Relevant Choices menu. 

The desired behavior Is to limit the speed to no more then 20 mph in the YELLOW state. To do this, the 
usertypes in a short program which implements the desired behavior. The program can be entered by clicking 
twice on the Action cell of this transition, causing Clsplay of a data entry window. 
*5 The program is entered into the ceil as follows: 

DriveJL-ight ( "Traf f ic_Ligb.t" , "Yellow" ) ; 
float desired_speed; 
50 desired_speed = POTEN_VAL; 

if (desirsdspeed > 20.0) desired_speed = 20.0; 
Drive_Dial{ "Speedometer" , desiredspeed ) ; 

ss 

Next, the user defines the response to the operator attempting to change the speed of the train. The user 
clicks on the EVENT cell of the second transition of t h e YELLOW state and selects t he Pfck-By-Name window 
select the Speed control. 
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The desired action for this cell isthe same as for the INITIALLY action. To copy the action, the user clicks 
on the ACTION cell of the INITIALLY transition of the state, dicks on the command COPY CELL, and then 
clicks on the empty action ceH below. To fill in the empty cell, the user clicks on the PASTE CELL command, 
causing the code to be pasted into the Action ceJL 

5 Finally, the user must describe the behavior if the user dicks on the changejight button while in the YEL- 

LOW state. To do this, the user clicks on the last cell in the EVENT column. From the Pick-By-Name window, 
the user clicks on the ChangeJJght window. From the relevant choices window, the user then clicks on the 
BUTTON event. Next, the user clicks on the last cell in the NEXT_STATE column. From the relevant choices 
menu, the user selects RED by clicking on the displayed term. 

10 Thie completes filling in the State Table, shown in completed form in Figure 30. 

The user must next save and compile the State Table. First the user clicks on the State Table Editor main 
pulldown menu. From this menu, the u3er selects the Logic Prototyping entry. This cause display of a flow 
chart like Logic Prototyping Control Panel. 

In the Logic Prototyping window, the user should click on the command SAVE. This instructs the State 

*5 Table Editor to save the State Table just entered into the warehouse. The user must give the State Table Edtor 
a name by typing in "TRAIN", Into the window which comes up, followed by pressing the <enter> key. The State 
Table is then saved. 

To compile the State Table, the user must click on the COMPILE ATN command in the Logic Prototyping 
window. Finally, when the MAKE TASKS command in the Logic Prototyping window is highlighted, the user 

» should cl it* on t he highlighted comma nd. This completes the behavioral part of the prototype. 

The final step is to run the Slate Table just created. To do this, the user Invokes the Runtime component 
of the prototype development system. This isdonebyaelectingthe Session command in the main State Table 
Editor pulldown window. Next, the user clicks on the command Run Thrt* Environment, initiating the Run 
Time components of the prototype development system. 

25 Using standard simulation system procedures, the user selects the warehouse where the compiled State 

Table b located. From the Storage Contents window, the user first selects the file TRAIN, with a file type of 
ATN, loads, and runs the file causing the prototype to begin execution. The user operates and interacts with 
t he prototype by, for example, changing the values cfthespeed control and pressing the button while watching 
the results on the speedometer and the traffic light. 

30 As demonstrated, the process of creating a State Table is relatively ample. Most of the requred informa- 

tion in a State Table can be entered with mouse clicks. When the default parameters that are entered into the 
State Table are not correct, the entries can be easily changed When necessary, the user can click twice on 
any field, and manually type in or edit the desired text 

In summary, the prototype development system according to the Invention provide* a system and method 

3s for defining the responses and states of a simulated system using a series of context sensitive menus to gen- 
erate state tables. The information from the state tables Is used to generatB program code used by the proto- 
type development platform or a target platform to si mulate a desired system. 

Although the present invention has been described and illustrated in detail, it is olearty understood that 
the same is by way of illustration and example only and is notto be taken byway of limitation, the spirit and 

« scope of the present invention being limited only by the terms of the appended claims. For example, although 
the State Table Editor has been described in the context of a system for simulating a physical system, the in- 
vention is also applicable toother systems such as real-time control, artificial intelligence, andotherdata proc- 
essing systems the operation of which can be defined by a state transition labia. 
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1. A real-time finite state prototype development and simulator system for graphically responding to user 
inputs by displaying virtual objects forming a modeled system on a monitor, comprising: 
so a graphic input device responsive to said user inputs for deriving a positional signal; and 

e processor unit including: 
(i) an input module responsive to said positional signal and to a position of one of the virtual objects 
for deriving selection data. 

fjl) an object editor module responsive to said selection data for defining ones of said virtual objects 
se as input an d output virtual objects and escribing to each a class of behavior, 

(iii) a state table editor module responsive to said selection data and to said class of behavior ascribed 
to said virtual objects for defining a table of parameters describing the modeled system including: 
[a) system states. 
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(b) events to be monitored, 

(c) actions to be taken in response to the monitored events, and 

(d) next states to be entered by said modeled system in response to the monitored events, and 
(iv) an execution module for detecting events end, responsive to said table of parameters and to said 
detected events, for selectively displaying and manipulating said virtual objects on the monitor. 

The system according to claim 1 wherein stale table editor module includes an interpreter module respon- 
sive to said table of parameters for supplying command data in the form of an Augmented Transition Net- 
work (ATN) Language. 

The system according to claim 1 wherein said stale table editor module includes a collection of reaction 
rules forming a reaction table stored in memory. 

A finite state machine for si mutating a system and responsive to a sign al repress nting a present state of 
the simulated system and to a plurality of Input signals for deriving an output signal Including a signal de- 
fining a next state of the simulated system, said finite state machine including; 

a random access memory responsive to an address signal for selectively retrieving instruction sig- 
nals stored therein; and 

a processing unit for storing the present simulated system state and responsive to said simulated 
system state, the plurality of input signals and said retrieved instruction signal forsupplylng said address 
signal ami defining the next state of the simulated system, said processing unll including 

(i) a state table editor module responsive to a user input signal for deriving a state table defining 

(a) present states of the simulated system. 

(b) input signals to be monitored by the simulated system, 

(c) actions to be taken by the simulated system in response to the monitored signals, and 

(iv) next states of the simulated system to be entered in response to the monitored signals, and 
fji) a runtime environment module responsive to said state labia far loading said instruction signals 
Into said random access memory. 

A method of real-time finite state simulation for graphically responding to user inputs by displaying virtual 
Objects on a montor, comprising the steps of: 

deriving a positional signal representing a position of a cursor on the monitor in response the user 

supplying selection data in response to (i) the positional signal representing the position of said 
cursor corresponding to (ii) a position of a virtual object on said monitor, 

defining virtual input and output type objects end associating with each a d ass of behavior respon- 
sive to said selection data; 

defining a state table in response to said selection date including: 

(i) states of a simulated system including the objects, 

(ii) simulated events to be monitored, 

{m) simulated actions to be taken by the simulated system in response to the monitored events, and 
fw) next States Of the Simulated system to be entered in response to the monitored events; and 
selectively displaying and manipulating said virtual objects in response to said state table, said 
selection data, other virtual objects and contents of variables. 

The method of claim 5 further comprising the step of associating with ones of said virtual objects respec- 
tive names of variables to which said associated class of behavior of said virtual objects are respectively 
responsive; 

A method of prototype development and prototype simulation, comprising the steps of: 
defining graphic objects for display on a monitor, 

associating with each of the graphic objects enumerated inputs to which each graphic object is 
responsive and a response of each graphic object to the respective enumerated Inputs; 

displaying a state table having plural rows and columns on the monitor; 

receiving a graphic device input signal and, in response, positioning and displaying a cursor on the 
monitor; 

detecting a position of said cursor corresponding to a position of one of said graphic objects on 
the monitor; 

displaying a menu of input and responses associated with said one graphic object; 
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selecting an Input and a response from the displayed menu: and 

entering data corresponding to the selected input and response in said state table. 

8. The method of claim 7 further comprising the step of selectively displaying the graphic objects on the 
monitor in response to input and response data stored in said state table. 

9. Amethod of graphically modelling a system, comprising the steps of: 

defining a set of graphic objects and attributing to each 

(i) at least one controlling parameter, 

(ii) a set of one or more displayable visual characteristics, and 

(iii) changeable features of said displayable visual characteristics responsive to said controlling para- 
meter 

describing a state table defining the operation of a finite state machine, the state table including 

(i) present states of the finite state machine, 

(ii) events to be monitored in each of the present states, 

(iii) controlling parameters to be generated in response to detection of a respective event to be moni- 
tored In said present states, and 

(iv) next states aft he finite state machinein response to detect ion of a respective event to be monitored 
in said present states; 

operating said finite state machine responsive to said state table to display said graphic objects 
including 

(i) defining an instant one of said present states of the finite state machine defined in sad state table. 

(ii) detecting an event of the events to be monitored in said instant present state, 

(Si) generating a controlling parameter in response to said detected event of said Instant present state 
in accordance with said state table, and 

(iv) defining a next state of the finite state machine in response to said detected event of said instant 
present state in accordance with said state table; and 

displaying said graphic objects in response to said controlling parameter generated by said gen- 
erating stBp. 

10. A system for developing and executing interactive visual applications wherein dynamic data is mapped 
into coherently animated virtual objects and operator interaction facilities are provided to interact with 
the virtual objects, the system comprising: 

a centra) processing unit responsive to operator control signals for supplying addrees and control 
signals; 

random access memory providing instructions to said central processing unit In response said ad- 
dress signals from said central processing unit; 

a non-volatile storage medium for supplying said instructions to said random access memory in 
response to said control signals from said central processing unit; 

a graphics display device responsive to said control signals from said central processing unit for 
displaying the virtual objects; and 

operator Input means for supplying said operator control signals to said central processing unit, 
said operator control signal including positional and alphanumeric signals, said operator input means in- 
cluding 

(1) a tocatorftrigger device lor providing said positional signals to the central processing unit, and 
(ii) a data entry device for supplying said alphanumeric signals to said central processing unit; 

said instructions including functional modules for defining operations of said central processing 
unit, including 

(i) an object editor module responsive to said operator control signals for defining the virtual objects 
by relating a graphical appearance of said virtual objects to a behavior of said objects during execution 
by the system; 

(ii) an integration editor module responsive to said operator control signals for mapping ones of the vir- 
tual objects into names of variables, the contents of mernot y locations of said random access memory 
corresponding to said named variables, said named variables being used for redrawing the virtual ob- 
jects on the graphics display to reflect changes in the contents of the memory locations associated 
with said named variables and said named variables 'urther supplying graphical access to data in re- 
sponse to said positional signals from said locator/trigger device and to screen locations of said virtual 
objects. 
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an execution facility for redrawing ones of said virtual objects in response to dynamic data stored 
»i said random access memory and In response to operator control signals, the execution facility fur- 
ther responsive to predetermined logic specification data; 

(iv) said logic specification data specifying sets of events and corresponding reactions which should 
occur in response to the respective events, an respective origin of the event, a state of the system 
whereinsaid state is afinitestate machine model of the system, and predetermined conditions, wtierein 
each of said events originating from one of (A) ones of said virtual objects, (B) a system dock signal, 
(C) control signals from said central processing unit, (D) messages from other computer systems, or 
(E) changes in the contents of the named variables: 

(v) said logic specification further specifying the next slate that the system should transition to after 



(vi) said logic specification further specifying the reactions that occur under contrd of the logio spec- 
if Icatlon, ones of which affect ones of the virtual objects displayed on the display device; and 

a state table editor responsive to said operator control signals for defining said logic specification 
by specifying: 
(I) system states; 

(ii) events and conditions to be recognized in respective ones of said States; 

(Hi) actions to be taken In response to respective ones of said events and states; and 

(iv) next states to be entered in response to respective ones of said events and states. 

11. The system of claim lOwhareln said logic specification is generated responsive toselectlon of the virtual 
objects displayed on the graphics display to f II in a state table including said events and conditions to 
be recognized and corresponding act ions to be taken. 

12. The system of claim 10 Including means tor specifying events and respective sources of said events by 
clicking with the ocator/triggar device on the visual representation of the virtual object generating respec- 
tive ones of said events. 

13. The system of claim 12 wherein said state table includes said logic specification data. 

14. The system of claim 1 3 Including means responsive to said locatorftijgger device for selectively inserting 
said logic specification data into said state table. 

15. The system of daim 11 including means responsive to said operator control signals for entering state 
names of respective ones of said states into the state table. 

16. The system of claim 15 wherein said means for entering said state n; 
from one of said cells forming said state tebie to another of said cells forming » 

1 7. The system of claim 1 9 wherein said execution facility indudes said logic specif icab'on and said execution 
facility is stored in said random access memory during a system execution mode of operation. 

1 8. The system of claim 10 in which said logic specification is stored in a computer controlled setup forming 
a real equivalent of a modeled system, wherein said computer controlled setup exhibits a behavior sub- 
stantially the same as a behavior of said modeled system. 
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ACTION PARAMETERS 
OblecC LIGHT/ TraflicUght 
Action: Change Stale 



Name Of State: | REDUGHT 
Figure 14 



ACTION PARAMETERS 


Object: XY OBJECT / Car Icon 
Event: Move Object 


Anchor 


R. ELATTVE JTO_CRT 


X Coordinate 


0.5 


Y Coordinate 


03 



Parameter value 
Held Name: NAME_0F_STATE 
Object: LKSHT/TRAFfTc LIGHT 
Type: STATE_NAME 



REDJJGHT 



GHEENJJGHT 



YELLOWJJGHT 



LEFT_TURN_UGHT 



WAU<JJGHT 



Parameter Value 
ReU Name: ANCHOR 
Object: XV OBJECT / Car Icon 
Type: ANCHORJTYP E 



FtELATTVE_TO_0RT 

RELATIVE_TO,FRAME 

ABSOLUTE 



SCALED_TO„FHAME 



Parameter Value 

Field Name X Coordinate Object: XY OBJECT / Car Icon Type: float 



Enter Value (0.0,3.0) 
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Examples of EVENTS for an object claw: 

Object. Clatm: KNOB 

KN0B.TUBN3SD 
KN0B_M0VBD 
KNOB^PHBSSKD 



Object Cla»«: FTELD 

FIELD VAUD 
FIELD INVALID 
FIELD 



Figure 21 



Examples of EVENTB that apply to no olgeet class: 

Object Class; NONE 

NO_DBVlC15 
ANYBVENT 
FLHBT^TIME 
INITIALLY 
IMMEDIATE 
FINAL 
COMMENT 
PERIODIC 
TIMEOUT 

USER 
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Examples of ACTIONS for an object class; 
Object Claw: LIGHT 

Light ChangeJState P Thi* changes the mod* Df the light. V 

o5«t_Ch4nge_Back_Golor /' This applisa to all objecte which havo a 

* background color. •/ 
Object-Move /* This applies to all movable objects. •/ 



Obi«t Claw : FIELD /* Tart fields can be input, output, or both in and out */ 

Object Chan«_Back .Color I* This applies to all object* which haw a 
~ * background color. *f 

/* This applies to all movable objects. */ 
/* This applies to all objecta that have text in 
♦ them. V 

P For a text field, aet its default value, */ 
/* Bet the validation rules for the field */ 
t* Bet the value of a field.*/ 
/* Set the value of the field U> the default 
•value.*/ 



Object.Movc 
Te*tJF , ont_Chang« 

Field_SetJ>ofault 
Field_9eLValidati6n 

Reld.S6t.Value 

KieleLResto«_Default 



Figure 2 3 



Examples of ACTIONS thai apply to no object class: 
Object ClaaB: NONE 

romom_number t* Compute a random number V 

prin Ldiapl ay /* Send » copy of the display to the printer */ 

set_eolorJi»;t '* Add ninnes to the color table. */ 

writ f* Stop all proceesing 7 i 

Figure 24 
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Figure 25 
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Figure 26 
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Figure 28 
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